I've built an application to import transactions into the database. Bad transactions go in a separate table and dupe transactions get updated. Currently, it takes about 2 hours to import ~40K records using the code below. Obviously I'd like this to run as fast as possible and since cursors are a real drag I was wondering if there was a more efficient way to accomplish this.
DECLARE
@.contact_id int,
@.product_code char(9),
@.status_date datetime,
@.business_code char(4),
@.expire_date datetime,
@.prod_status char(4),
@.transaction_id int,
@.emailAddress varchar(50),
@.journal_id int
BEGIN TRAN
DECLARE transaction_import_cursor CURSOR
FOR SELECT transaction_id, product_code, emailAddress, status_date, business_code, expire_date, prod_status from transactions_batch_tmp
OPEN transaction_import_cursor
FETCH NEXT FROM transaction_import_cursor INTO @.transaction_id, @.product_code, @.emailAddress, @.status_date, @.business_code, @.expire_date, @.prod_status
WHILE (@.@.FETCH_STATUS = 0)
BEGIN
SELECT top 1 contacts.contact_id AS contact_id, transactions_batch_tmp.status_date AS status_date, transactions_batch_tmp.product_code AS product_code,
transactions_batch_tmp.business_code AS business_code, transactions_batch_tmp.expire_date AS expire_date,
transactions_batch_tmp.prod_status AS product_status
FROM transactions_batch_tmp INNER JOIN
journal INNER JOIN
contacts ON journal.contact_id = contacts.contact_id ON transactions_batch_tmp.emailAddress = contacts.emailAddress AND
transactions_batch_tmp.product_code = journal.product_code INNER JOIN
products ON transactions_batch_tmp.product_code = products.product_code
WHERE rtrim(ltrim(contacts.emailAddress)) = @.emailAddress AND journal.product_code = @.product_code
ORDER BY transactions_batch_tmp.status_date desc
IF @.@.ROWCOUNT = 0
BEGIN
print 'NEW transaction! ' + @.product_code + @.emailAddress
insert into journal (contact_id, product_code, status_date, business_code, expire_date, entryTypeID, product_status, date_entered)
SELECT distinct rtrim(ltrim(contacts.contact_id)) as cid, rtrim(ltrim(products.product_code)), transactions_batch_tmp.status_date,
rtrim(ltrim(transactions_batch_tmp.business_code)) , transactions_batch_tmp.expire_date, 21, rtrim(ltrim(transactions_batch_tmp.prod_status)), getDate()
FROM contacts INNER JOIN (transactions_batch_tmp INNER JOIN products ON transactions_batch_tmp.product_code=products.produ ct_code) ON contacts.emailAddress=transactions_batch_tmp.email Address
WHERE transactions_batch_tmp.transaction_id=@.transaction _id
END
ELSE
BEGIN
--print 'UPDATE transaction! ' + @.product_code + @.emailAddress
UPDATE journal
SET status_date =
(SELECT max(tmp.status_date)
FROM transactions_batch_tmp tmp, contacts c, products p, journal j
WHERE tmp.emailaddress = @.emailAddress
AND tmp.emailaddress = rtrim(c.emailaddress)
AND c.contact_id = j.contact_id
AND j.product_code = @.product_code
AND j.product_code = tmp.product_code)
FROM transactions_batch_tmp tmp, contacts c, products p, journal j
WHERE tmp.emailaddress = @.emailAddress
AND tmp.emailaddress = rtrim(c.emailaddress)
AND c.contact_id = j.contact_id
AND j.product_code = @.product_code
AND j.product_code = tmp.product_code
END
FETCH NEXT FROM transaction_import_cursor INTO @.transaction_id, @.product_code, @.emailAddress, @.status_date, @.business_code, @.expire_date, @.prod_status
END
CLOSE transaction_import_cursor
DEALLOCATE transaction_import_cursor
COMMIT TRAN
/** purge data from temp error table before writing bad records for this batch **/
truncate table tran_import_error;
/** write bad records (missing product code or email address) to temp_error table **/
insert into tran_import_error (transaction_id, product_code, emailAddress, date_entered)
SELECT DISTINCT transactions_batch_tmp.transaction_id, transactions_batch_tmp.product_code, transactions_batch_tmp.emailAddress, getDate()
FROM transactions_batch_tmp
where transactions_batch_tmp.emailaddress not in (select emailaddress from contacts)
OR
transactions_batch_tmp.product_code not in (select product_code from products)
TIAI don't see anything in your code that requires a cursor. It would run much faster as set-based INSERT and UPDATE statements.|||Well, how would I handle the update part without a cursor? I need to make sure that *only* unique contact_id-product_code values exist in the journal table.
Thanks.|||Add some bit flag and notes columns to your import table. Then you can run data checks against the records prior to importing them. Flag any duplicates or bad records and add a note as to why they were flagged. Then import only the non-flagged records. Delete the non-flagged records when you are done, and you are left with a list of bad records that you can review or discard.|||blindman - Thanks for your help.
I'm almost there (I hope), but was wondering if there was a more efficient way to delete the dupe records than having to write two separate queries. I need to keep the most recent product_code-status_date transaction for *each* person. This runs after I insert ALL the records in the journal table.
--delete dupe trans with status_date as the flag
DELETE journal FROM journal, contacts
JOIN
(select product_code, contact_id, max(status_date) as max_status_date
from journal
group by product_code, contact_id) AS G
ON G.[contact_id] = contacts.[contact_id]
WHERE journal.[status_date] < G.[max_status_date]
AND G.[product_code] = journal.[product_code];
--delete dupe trans with journal_id as the flag
DELETE journal FROM journal, contacts
JOIN
(select product_code, contact_id, max(journal_id) as maxID
from journal
group by product_code, contact_id) AS G
ON G.[contact_id] = contacts.[contact_id]
WHERE journal.[journal_id] < G.[MaxID]
AND G.[product_code] = journal.[product_code];
Thanks again.|||The contact table has nothing to do with your delete, except to limit the deleted records to those that have a contact_id. I assume that contact_id is part of journal's natural key and that all records have a valid contact_id, so drop if from both your queries. (If you do need it for filtering, join it in the subquery.)
--delete dupe trans with status_date as the flag
DELETE
FROM journal
INNER JOIN
(select product_code, contact_id, max(status_date) as max_status_date
from journal
group by product_code, contact_id) AS G
ON journal.[product_code] = G.[product_code]
and journal.[contact_id] = G.[contact_id]
and journal.[status_date] < G.[max_status_date]
--delete dupe trans with journal_id as the flag
DELETE journal
FROM journal
INNER JOIN
(select product_code, contact_id, max(journal_id) as maxID
from journal
group by product_code, contact_id) AS G
ON journal.[product_code] = G.[product_code]
and journal.[contact_id] = G.[contact_id]
and journal.[journal_id] < G.[MaxID]
It also appears that the first query should handle all product_code/contact_id duplicates except those with that share exactly the same status_date. If status_date stores only whole-date values, then I guess I see the point of the second delete statement, but otherwise I wouldn't expect you to get a high rowcount from it.
Now to your question; can this be done as a single SQL statement? Yes, but it would essentially require two nested subqueries, so I don't think you would get a big performance boost from it, and you would certainly have to sacrifice code clarity. I recommend that you leave it as two separate deletes.
Showing posts with label built. Show all posts
Showing posts with label built. Show all posts
Tuesday, March 27, 2012
Thursday, March 22, 2012
Do I need a data extension?
I'm working on a project for a national restaurant chain. I've built a
complex class that allows them to do various forms of sales estimations for
stores who fail to report their weekly sales. I say "complex" because it
has a detailed class hierarchy under it and performs lot of SQL Server work
inside its "black box." Using this class within a web page is working
great...
Now they want this functionality to also appear in a report, and I'm
wondering what I have to do -- never done a RS data extension and first am
wondering: Is this what I need to add this function to a report?
If the answer is "yes," is there a resource that describes in detail how to
do it, with a non-trivial example?
I'm really under the gun on this one, so I'll be grateful for any help --
thanks!
JoeI'd start with the sample reports that ship with the product. Check
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/RSAMPLES/htm/rss_overview_v1_631v.asp?frame=true
for details.
--
Ravi Mumulla (Microsoft)
SQL Server Reporting Services
This posting is provided "AS IS" with no warranties, and confers no rights.
"Joe Helmick" <joe_helmick@.4dv.net> wrote in message
news:O2r7tE5jEHA.1652@.TK2MSFTNGP09.phx.gbl...
> I'm working on a project for a national restaurant chain. I've built a
> complex class that allows them to do various forms of sales estimations
for
> stores who fail to report their weekly sales. I say "complex" because it
> has a detailed class hierarchy under it and performs lot of SQL Server
work
> inside its "black box." Using this class within a web page is working
> great...
> Now they want this functionality to also appear in a report, and I'm
> wondering what I have to do -- never done a RS data extension and first am
> wondering: Is this what I need to add this function to a report?
> If the answer is "yes," is there a resource that describes in detail how
to
> do it, with a non-trivial example?
> I'm really under the gun on this one, so I'll be grateful for any help --
> thanks!
> Joe
>|||Joe,
One option in your scenario may be to dump your object data (state) in a
database before the report is run. Unfortunately, version 1.0 of Reporting
Services doesn't support events, so to implement preprocessing you need to
get somewhat innovative. One option is to attach an expression to a property
of the report body band (it will fire before the regions get loaded). This
expression could gather the report parameters, pass it to your object which
in turn can serialize itself to the database and let the regions pick data
from there. Of course, this is a rather simplified version of what you may
need to do.
Yes, another and more elegant option is to wrap you object in a custom data
extension. This is actually very straightforward. You need to expose your
object data as a forward-only tabular stream which conforms to the
IDataReader interface. You may find my ADO.NET custom data extension
(http://www.gotdotnet.com/Community/UserSamples/Details.aspx?SampleGuid=B846
8707-56EF-4864-AC51-D83FC3273FE5) useful to get you started. It comes with
step-by-step help instructions.
Hope this helps.
----
Teo Lachev, MCSD, MCT
Author: "Microsoft Reporting Services in Action"
Publisher website: http://www.manning.com/lachev
Buy it from Amazon.com: http://shrinkster.com/eq
Home page and blog: http://www.prologika.com/
----
"Joe Helmick" <joe_helmick@.4dv.net> wrote in message
news:0l1aj0h4452vd72jles53foioiqfhfdphn@.4ax.com...
> Ravi,
> Thanks, but I have all the samples and unless I missed something due
> to my newbie-ness, I don't see what I need. I don't need drilldown or
> drillthrough, and simple functions in a code-behind don't seem like
> they'd nearly fill the bill. I'm already doing cascading parameters
> from multiple datasets and stuff like that, no problem, but attaching
> functionality from a 700-line VB class with deep inheritance? I don't
> see anything like that in the samples...
> Did I miss something, or do I need another resource besides the
> samples?
> Joe
> "Ravi Mumulla \(Microsoft\)" <ravimu@.online.microsoft.com> bellowed
> forth with this wisdom for all to hear:
> > I'd start with the sample reports that ship with the product. Check
> >
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/RSAMPLES/htm/rss_overview_v1_631v.asp?frame=true
> > for details.
>sql
complex class that allows them to do various forms of sales estimations for
stores who fail to report their weekly sales. I say "complex" because it
has a detailed class hierarchy under it and performs lot of SQL Server work
inside its "black box." Using this class within a web page is working
great...
Now they want this functionality to also appear in a report, and I'm
wondering what I have to do -- never done a RS data extension and first am
wondering: Is this what I need to add this function to a report?
If the answer is "yes," is there a resource that describes in detail how to
do it, with a non-trivial example?
I'm really under the gun on this one, so I'll be grateful for any help --
thanks!
JoeI'd start with the sample reports that ship with the product. Check
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/RSAMPLES/htm/rss_overview_v1_631v.asp?frame=true
for details.
--
Ravi Mumulla (Microsoft)
SQL Server Reporting Services
This posting is provided "AS IS" with no warranties, and confers no rights.
"Joe Helmick" <joe_helmick@.4dv.net> wrote in message
news:O2r7tE5jEHA.1652@.TK2MSFTNGP09.phx.gbl...
> I'm working on a project for a national restaurant chain. I've built a
> complex class that allows them to do various forms of sales estimations
for
> stores who fail to report their weekly sales. I say "complex" because it
> has a detailed class hierarchy under it and performs lot of SQL Server
work
> inside its "black box." Using this class within a web page is working
> great...
> Now they want this functionality to also appear in a report, and I'm
> wondering what I have to do -- never done a RS data extension and first am
> wondering: Is this what I need to add this function to a report?
> If the answer is "yes," is there a resource that describes in detail how
to
> do it, with a non-trivial example?
> I'm really under the gun on this one, so I'll be grateful for any help --
> thanks!
> Joe
>|||Joe,
One option in your scenario may be to dump your object data (state) in a
database before the report is run. Unfortunately, version 1.0 of Reporting
Services doesn't support events, so to implement preprocessing you need to
get somewhat innovative. One option is to attach an expression to a property
of the report body band (it will fire before the regions get loaded). This
expression could gather the report parameters, pass it to your object which
in turn can serialize itself to the database and let the regions pick data
from there. Of course, this is a rather simplified version of what you may
need to do.
Yes, another and more elegant option is to wrap you object in a custom data
extension. This is actually very straightforward. You need to expose your
object data as a forward-only tabular stream which conforms to the
IDataReader interface. You may find my ADO.NET custom data extension
(http://www.gotdotnet.com/Community/UserSamples/Details.aspx?SampleGuid=B846
8707-56EF-4864-AC51-D83FC3273FE5) useful to get you started. It comes with
step-by-step help instructions.
Hope this helps.
----
Teo Lachev, MCSD, MCT
Author: "Microsoft Reporting Services in Action"
Publisher website: http://www.manning.com/lachev
Buy it from Amazon.com: http://shrinkster.com/eq
Home page and blog: http://www.prologika.com/
----
"Joe Helmick" <joe_helmick@.4dv.net> wrote in message
news:0l1aj0h4452vd72jles53foioiqfhfdphn@.4ax.com...
> Ravi,
> Thanks, but I have all the samples and unless I missed something due
> to my newbie-ness, I don't see what I need. I don't need drilldown or
> drillthrough, and simple functions in a code-behind don't seem like
> they'd nearly fill the bill. I'm already doing cascading parameters
> from multiple datasets and stuff like that, no problem, but attaching
> functionality from a 700-line VB class with deep inheritance? I don't
> see anything like that in the samples...
> Did I miss something, or do I need another resource besides the
> samples?
> Joe
> "Ravi Mumulla \(Microsoft\)" <ravimu@.online.microsoft.com> bellowed
> forth with this wisdom for all to hear:
> > I'd start with the sample reports that ship with the product. Check
> >
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/RSAMPLES/htm/rss_overview_v1_631v.asp?frame=true
> > for details.
>sql
Sunday, March 11, 2012
dll in c# sql 2005
I wrote a class in C# with one method that returns a string. I built the dll and deployed it into sql 2005 and everything works just fine.
I am now in a situation where I need to use this function in SQL 2000. Is there any way that I can use the existing dll and create a sql 2000 function, and if so how do I go about doing this?
Here is the c# code from the deployed dll.
using System;
using System.Data;
using System.Data.SqlClient;
using System.Data.SqlTypes;
using Microsoft.SqlServer.Server;
using System.Text.RegularExpressions;
public partial class GECBI
{
[Microsoft.SqlServer.Server.SqlFunction(IsDeterministic = true, IsPrecise = true)]
public static string RegExMatch(string pattern, string matchString)
{
Regex r1 = new Regex(pattern);
if (r1.Match(matchString.TrimEnd(null)).Success)
{
Regex testReg = new Regex(pattern);
Match regMatch = testReg.Match(matchString);
string runnumber = regMatch.ToString();
return runnumber;
}
else
{
return null;
}
}
};
Accesing CLR Function is only possible in SQL2k5. Wrapping the .NET object in a COM object or calling the function on a SQL2k5 Server would be very slow during execution time.HTH, JEns K. Suessmeyer.
http://www.sqlserver2005.de|||So what would be an alternative for me to achieve the above? Can someone give me an example of how to create the above function in something that I can use with sql 2k please?|||There is actually no alternativ for that. have a look in the internet, there have been some samples implemented to mimic the functionality.
http://www.google.de/search?hl=de&q=regular+expressions+sql+server+2000&meta= http://www.codeproject.com/managedcpp/xpregex.asp
HTH, Jens K. Suessmeyer.
http://www.sqlserver2005.de
DLINQ to DataTable ?
Is there a built in way of converting a LINQ qurey result into a DataTable or DataView ?
The LINQ query result is a collection of strongly typed objects. There is no built-in method to do what you want,although it is possible.
|||
I know it is possibe to do this via reflection, however it could be problematic when using anonymous types.
Anyway I was looking for something built-in and I guess there is none which is a shame. Beta versions of DLINQ did have a built in mechanism.
|||Sorry, I spoke way too soon:http://blogs.msdn.com/erickt/archive/2007/10/24/linq-to-dataset-data-binding-linqdataview-restriction-joins.aspx
AsDataView() would appear to be what you are after.
|||
AsDataView() is not a part of the release version. From what I've read so far it was available in pervious Beta versions. It is only avaiable as a DataTable extension method and it can be used with Linq to DataSet andnot with Linq to SQL.
Wednesday, March 7, 2012
Distribution without Exchange.
Is it possible to use the automatic distribution that is built into Reporting
Services 2005 if you do not have access to an Exchange server. We currently
outsource our email and would like to keep away from having to put Exchange
within our network.
Is this possible? If so, can someone point me to the proper place to begin
looking.
ThanksI could be very mistaken, but I don't SSRS requires Exchange. It only need an
SMTP Server...
--
Wayne Snyder MCDBA, SQL Server MVP
Mariner, Charlotte, NC
I support the Professional Association for SQL Server ( PASS) and it''s
community of SQL Professionals.
See you at the PASS Conference!
"dgator" wrote:
> Is it possible to use the automatic distribution that is built into Reporting
> Services 2005 if you do not have access to an Exchange server. We currently
> outsource our email and would like to keep away from having to put Exchange
> within our network.
> Is this possible? If so, can someone point me to the proper place to begin
> looking.
> Thanks|||On Feb 13, 3:26 pm, Wayne Snyder <wayne.nospam.sny...@.mariner-usa.com>
wrote:
> I could be very mistaken, but I don't SSRS requires Exchange. It only need an
> SMTP Server...
> --
> Wayne Snyder MCDBA, SQL Server MVP
> Mariner, Charlotte, NC
> I support the Professional Association for SQL Server ( PASS) and it''s
> community of SQL Professionals.
> See you at the PASS Conference!
> "dgator" wrote:
> > Is it possible to use the automatic distribution that is built into Reporting
> > Services 2005 if you do not have access to an Exchange server. We currently
> > outsource our email and would like to keep away from having to put Exchange
> > within our network.
> > Is this possible? If so, can someone point me to the proper place to begin
> > looking.
> > Thanks
You are correct. This article also eludes to it.
http://www.microsoft.com/technet/prodtechnol/sql/2005/usingssrswithsqlexpress.mspx
Regards,
Enrique Martinez
Sr. Software Consultant
Services 2005 if you do not have access to an Exchange server. We currently
outsource our email and would like to keep away from having to put Exchange
within our network.
Is this possible? If so, can someone point me to the proper place to begin
looking.
ThanksI could be very mistaken, but I don't SSRS requires Exchange. It only need an
SMTP Server...
--
Wayne Snyder MCDBA, SQL Server MVP
Mariner, Charlotte, NC
I support the Professional Association for SQL Server ( PASS) and it''s
community of SQL Professionals.
See you at the PASS Conference!
"dgator" wrote:
> Is it possible to use the automatic distribution that is built into Reporting
> Services 2005 if you do not have access to an Exchange server. We currently
> outsource our email and would like to keep away from having to put Exchange
> within our network.
> Is this possible? If so, can someone point me to the proper place to begin
> looking.
> Thanks|||On Feb 13, 3:26 pm, Wayne Snyder <wayne.nospam.sny...@.mariner-usa.com>
wrote:
> I could be very mistaken, but I don't SSRS requires Exchange. It only need an
> SMTP Server...
> --
> Wayne Snyder MCDBA, SQL Server MVP
> Mariner, Charlotte, NC
> I support the Professional Association for SQL Server ( PASS) and it''s
> community of SQL Professionals.
> See you at the PASS Conference!
> "dgator" wrote:
> > Is it possible to use the automatic distribution that is built into Reporting
> > Services 2005 if you do not have access to an Exchange server. We currently
> > outsource our email and would like to keep away from having to put Exchange
> > within our network.
> > Is this possible? If so, can someone point me to the proper place to begin
> > looking.
> > Thanks
You are correct. This article also eludes to it.
http://www.microsoft.com/technet/prodtechnol/sql/2005/usingssrswithsqlexpress.mspx
Regards,
Enrique Martinez
Sr. Software Consultant
Saturday, February 25, 2012
Distribution Cost
Sorry if this is the wrong place to post this.....
I am new to MS SQL as I have been using Paradox for years. Paradox has its own built in DB, so, the cost is great for small company applications (which is what we do). However, we are now going to rewrite our application in VB (most likely), and we want to ensure that the DB we go with will not cost a fortune to our clients. On top of this we want the application to be PC based as well as online. I know you can connect to MS SQL via PHP. The DB will live on our server, and the clients should all be hosted on our server. Our clients are Churches, and they do not have thousands to put into a program, which is why Paradox was great for this. Now, I am worried that MS SQL will cost for every copy of the program we sale. How does this work if we develop a program to work with MS SQL 2005 and sale it to clients? Is there a cost that we must pay MS for each copy? Does the client need the MS SQL Desktop Engine to run the DB, or do they need something else?
Sorry to sound stupid, but MS SQL is new to me, and I want to ensure that I make the right choice that is not going to bite me later down the road. I have looked at MySQL, PostgreSQL, OpenBase, and a few others. The all seem ok, but still looking. Thanks in advance for any suggestions or comments you have...
Gene
Hi Gene,
Perhaps SQL Server Express edition will meet your business needs. Check out http://www.microsoft.com/sql/editions/express/default.mspx for details. You may also find this topic useful for comparing the features available in each edition: http://www.microsoft.com/sql/prodinfo/features/compare-features.mspx and this site contains useful information about pricing and licensing http://www.microsoft.com/sql/howtobuy/default.mspx.
Regards,
Gail
Subscribe to:
Posts (Atom)