Friday, 13 April 2012

File Exists Check in SSIS

Couple of weeks back I was working with a SSIS package development. My requirement is to check whether the file exists before I start importing the data. If it doesn’t exist then the package should update the db table. When I heard this requirement I thought I can make use of the “File System Task” in SSIS toolbox. When I start the work I came to know that you can’t get this piece of information from that task. It’s really weird for me as you can move\copy a file or folder using this task however checking the file existence is not possible with that task as shown below.

file_exists_ssis_1
So I need to do some workaround to get rid of my requirement. There are couple of methods which I’ve tried and both worked like a charm. Below are the two ways you can attain the requirement
  • Using “Script Task”
  • Using “File Properties Task”, this is not a default task. This is developed in codeplex and it’s shared for free. You need to install it to add the user defined task in SSIS. You can download it from the link http://filepropertiestask.codeplex.com/
Let’s test both the ways
Using “Script Component Task”
In this method I’ll use script component task. I’ll use dot net scripting inside this script task to check the existence of the file. To use this method, drag and drop “Script Task” into the package. I’m declaring two variables (one as string and result variable as Boolean)to the package, first variable is to point the file and second one is to return the status. Use these two variables as shown below and click on Edit script button
*) Provide file path variable as readonly variable and provide another variable to get the result. In this example Filepath variable contains the file path and Result variable has the result of file existence
file_exists_ssis_2
*) Now click on Edit script , you will be poped up with Visual studio editor. You can see from the image below I’ve added System.IO name space which is used to get file properties. Once you added the name space scroll down till you see Public main, there add the code below to check the file existence and return the value a variable
1
Dts.Variables("Result").Value = File.Exists(Dts.Variables("FilePath").Value.ToString)
file_exists_ssis_3
*) After add the code click OK on script task. In my example I need to execute import task if the file exists or else it need to update SQL Server db table with details. Below is the package I developed, if you can see there is a function symbol displayed in the direction flow, that’s where I’m checking the file existence
file_exists_ssis_5
*) Double click on the flow path and choose Evaluation Operation as “Expression and Constraint” and then in Expression provide the variable name where we get the file existence details. From the image you can see I’ve set it to true so that when file exists it will use that path.
file_exists_ssis_4
That’s it all set. Now you have used script task to check file existence status.
I’ve faced the below error while using the script task.
Namespace or type specified in the Imports ‘Microsoft.SqlServer.Dts.Runtime’ doesn’t contain any public member or cannot be found. Make sure the namespace or the type is defined and contains at least one public member. Make sure the imported element name doesn’t use any aliases.
Script task uses “Imports Microsoft.SqlServer.Dts.Runtime” name space which is not getting resolved. When I checked I couldn’t find the dll “Microsoft.SqlServer.Dts.Design.dll” in the directory “C:\Program Files\Microsoft SQL Server\100\SDK\Assemblies\” which means I haven’t installed SDK. So to resolve this make sure you have installed “Client Tools SDK” during SQL Server installation as shown below
file_exists_ssis_6
Using “File Properties Task”
As I said earlier this isn’t a default task. You need to install it, once it’s installed you can see it in the toolbox. Just drag and drop the task into the package and open the task.
file_exists_ssis_7
You need to provide the source path. In this case I stored it in a variable so I pointed it to FilePath variable. Next you need to click on the drop down menu and select “Task always succeeds and records file existence in variable”, once you chosen you will be able to select the destination variable, In this example I’ve chosen the Result variable. Now do the same logic which we applied in the control flow which is just to add this expression. That’s all, all set. This method is pretty straight forward you don’t need to type your code instead you can directly get the result to a variable. I prefer this method however in some organizations they won’t allow us to use the open source references so in that case we can go for the first method.
In addition to this  you can also accomplish this with the below tasks however I prefer to stay with these two methods.
  • Foreach Loop Container Task– Just provide the filename explicitly in the path
  • Execute SQL Task – You can use XP_FileExists to get the status.
I had a curiosity to check whether Microsoft has added this small piece of code in File system task on SQL Server 2012. Do you know this is not added in SQL Server 2012 Integration services, so  you are left only with these methods Smile . I don’t think MS will add it to the “File System Task”, there is a suggestion already raised in Microsoft Connect (http://connect.microsoft.com/SQLServer/feedback/details/231838/check-if-file-exists-operation-in-ssis-file-system-task) but they closed it.

Wednesday, 14 March 2012

Sql Server Resource Fails to come online on Cluster with Error Error Locating Server/Instance Specified [xFFFFFFFF]

issue : unable to start SQL server cluster instance and got it following error message.

Error message:

Sql Server Resource Fails to come online on Cluster with Error Error Locating Server/Instance Specified [xFFFFFFFF]

CAUSE =====
So we checked Sql browser service and found that the sql browser service was stopped and was in manual mode.
Sql Browser is the one which is responsible to route all the incoming connections for the sql server to their appropriate TCP port on which the sql server is listening. Sql browser service itself listen on port 1434 on UDP protocol. So whenever a client attempts to connect to the named instance of sql server, it will first try to connect to the Sql browser service which runs on port 1434 and then routes to the appropriate port on which sql server instance is listening.
However the above explanation does not hold true for default instance of the sql server which runs on port 1433 and hence the client does not require sql browser to route it’s connection but instead connects directly to port.

RESOLUTION
=========

So we started the sql browser service and changed it state to Automatic from manual. We observed that sql server came online fine without any error on node 2 without any error.

Wednesday, 29 February 2012

unable to start the SQL server and getting error message "Win32 or C++ exception, or by an access violation encountered during exception handling"

 SQL Error :
Event Type:        Error
Event Source:    MSSQL$VSDBW07
Event Category:                (2)
Event ID:              17311
Date:                     9/18/2011
Time:                    2:10:56 AM
User:                     N/A
Computer:          HQRDCA0068
Description:
SQL Server is terminating because of fatal exception c0000005. This error may be caused by an unhandled Win32 or C++ exception, or by an access violation encountered during exception handling. Check the SQL error log for any related stack dumps or messages. This exception forces SQL Server to shutdown. To recover from this error, restart the server (unless SQLAgent is configured to auto restart).

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Cause of the issue : Problem was due to .NET Framework v2.0 corruption.

Resoultion:
1)   In folder C:\WINNT\Microsoft.NET\Framework  rename folder  v2.0.50727  to tmp_v2.0.50727
        In folder C:\WINNT\Microsoft.NET\Framework64  rename folder v2.0.50727  to tmp_v2.0.50727

2) Since removal of .NET framework from control panel was not possible (You should verify)
We followed this kb article:  http://support.microsoft.com/kb/908077

3) Launch SQL server install for SQL engine and perform the first steps until .NET framework got installed then cancel the installation
4) Check and ensure the folder v2.0.50727  got created in C:\WINNT\Microsoft.NET\Framework64
5) Launch SQL server install for SQL tools and perform the first steps until .NET framework got installed then cancel the installation
6) Check and ensure the folder v2.0.50727  got created in C:\WINNT\Microsoft.NET\Framework 
7) Server restart
8) Test
9) Remove folder tmp_v2.0.50727 from C:\WINNT\Microsoft.NET\Framework  and C:\WINNT\Microsoft.NET\Framework64

Robocopy (Robust File and Folder Copy)

What is Robocopy ?

Robocopy, or "Robust File Copy", is a command-line directory replication
command.  Normally it used to copy folders and files from one location to another location without network error.   Also it is bit fast when compare with normal copy.


Syntax :
robocopy "\\SERVERNAME\G$\MS_SQLServer\SQL2005\9000\Latest SP4 Express and hotfix" "F:\Software" /E

Source options
                /S : Copy Subfolders.
                /E : Copy Subfolders, including Empty Subfolders.

script to find the rowcounts of all the tables in a database

script to find the rowcounts of all the tables in a database


SELECT o.name, i.[rows]
FROM sysobjects o
INNER JOIN sysindexes i
ON  o.id = i.id
WHERE (o.type = 'u') AND (i.indid = 1)
ORDER BY o.name

Enable xp_cmdshell

<><><><>
Enable the xp_cmdshell
---------------------------
use master
EXEC master.dbo.sp_configure 'show advanced options', 1
Go
RECONFIGURE with override
Go
EXEC master.dbo.sp_configure 'xp_cmdshell', 1
Go
RECONFIGURE with override
Go
EXEC master.dbo.sp_configure 'show advanced options', 0
Go
RECONFIGURE with override
---------------------------

Unable to uninstall SQL Server fully from control panel

When we tried to uninstall the SQL server however service not went from windows service.

Resolution:

delete windows service from command line:

Syntax :

sc delete <service-name>

SQL Server Patching Error message : "This installation is forbidden by system policy. Contact your system administrator."


Error message :
 

Info 1625.This installation is forbidden by system policy.  Contact your system administrator.
f:\948292ea6c815ff61807b43e22\setup\sqlncli.msi
MSI (s) (A4:F4) [07:00:06:983]: User policy value 'DisableRollback' is 0
MSI (s) (A4:F4) [07:00:06:983]: Machine policy value 'DisableRollback' is 0
MSI (s) (A4:F4) [07:00:06:983]: Incrementing counter to disable shutdown. Counter after increment: 0
MSI (s) (A4:F4) [07:00:06:983]: Note: 1: 1402 2: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Installer\Rollback\Scripts 3: 2
MSI (s) (A4:F4) [07:00:06:983]: Note: 1: 1402 2: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Installer\Rollback\Scripts 3: 2
MSI (s) (A4:F4) [07:00:06:983]: Note: 1: 1402 2: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Installer\InProgress 3: 2
MSI (s) (A4:F4) [07:00:06:983]: Note: 1: 1402 2: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Installer\InProgress 3: 2
MSI (s) (A4:F4) [07:00:06:983]: Decrementing counter to disable shutdown. If counter >= 0, shutdown will be denied.  Counter after decrement: -1



Cause : This error message occuring due to "security policy issue on this server." Policy not allowing sql server to install the windows installer.




Resoultion : Change the registry key like below to resolve this patching issue.

In this case you need modify the registry key : HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\Installer and modify the value of the key DisableMSI from 2 to 0
Don’t forget to set it back to 2 after the install


Info 1625.This installation is forbidden by system policy.  Contact your system administrator.”

How to Change Maintenance plan job owner into sa


How to Change Maintenance plan job owner into sa  :

EXEC MSDB.dbo.sp_update_job
@job_name = 'DailyBackups',
@owner_login_name = 'sa'
GO

Saturday, 25 February 2012

SQL SERVER – 2005 – Start Stop Restart SQL Server From Command Prompt

Very frequently I use following command prompt script to start and stop default instance of SQL Server. Our network admin loves this commands as this is very easy.
Click Start >> Run >> type cmd to start command prompt.
Start default instance of SQL Server
net start mssqlserver
Stop default instance of SQL Server
net stop mssqlserver
Start and Stop default instance of SQL Server.
You can create batch file to execute both the commands together.

Friday, 24 February 2012

SQL SERVER – Fragmentation – Detect Fragmentation and Eliminate Fragmentation

What is Fragmentation? How to detect fragmentation and how to eliminate it?
Storing data non-contiguously on disk is known as fragmentation. Before learning to eliminate fragmentation, you should have a clear understanding of the types of fragmentation. We can classify fragmentation into two types:
  • Internal Fragmentation: When records are stored non-contiguously inside the page, then it is called internal fragmentation. In other words, internal fragmentation is said to occur if there is unused space between records in a page. This fragmentation occurs through the process of data modifications (INSERT, UPDATE, and DELETE statements) that are made against the table and therefore, to the indexes defined on the table. As these modifications are not equally distributed among the rows of the table and indexes, the fullness of each page can vary over time. This unused space causes poor cache utilization and more I/O, which ultimately leads to poor query performance.
  • External Fragmentation: When on disk, the physical storage of pages and extents is not contiguous. When the extents of a table are not physically stored contiguously on disk, switching from one extent to another causes higher disk rotations, and this is called Extent Fragmentation.
Index pages also maintain a logical order of pages inside the extent. Every index page is linked with previous and next page in the logical order of column data. However, because of Page Split, the pages turn into out-of-order pages. An out-of-order page is a page for which the next physical page allocated to the index is not the page pointed to by the next-page pointer in the current leaf page. This is called Logical Fragmentation.
Ideal non-fragmented pages are given below:
Statistics for table scan are as follows:
  • Page read requests: 2
  • Extent switches: 0
  • Disk space used by table: 16 KB
  • avg_fragmentation_in_percent: 0
  • avg_page_space_used_in_percent: 100
Following are fragmented pages:
In this case, the statistics for table scan are as follows:
  • Page read requests: 6
  • Extent switches: 5
  • Disk space used by table: 48 KB
  • avg_fragmentation_in_percent > 80
  • avg_page_space_used_in_percent: 33
How to detect Fragmentation: We can get both types of fragmentation using the DMV: sys.dm_db_index_physical_stats. For the screenshot given below, the query is as follows:
SELECT OBJECT_NAME(OBJECT_ID), index_id,index_type_desc,index_level,
avg_fragmentation_in_percent,avg_page_space_used_in_percent,page_count
FROM sys.dm_db_index_physical_stats
(DB_ID(N'AdventureWorksLT'), NULL, NULL, NULL , 'SAMPLED')
ORDER BY avg_fragmentation_in_percent DESC
Along with other information, there are two important columns that for detecting fragmentation, which are as follows:
  • avg_fragmentation_in_percent: This is a percentage value that represents external fragmentation. For a clustered table and leaf level of index pages, this is Logical fragmentation, while for heap, this is Extent fragmentation. The lower this value, the better it is. If this value is higher than 10%, some corrective action should be taken.
  • avg_page_space_used_in_percent: This is an average percentage use of pages that represents to internal fragmentation. Higher the value, the better it is. If this value is lower than 75%, some corrective action should be taken.
Reducing fragmentation:
  • Reducing Fragmentation in a Heap: To reduce the fragmentation of a heap, create a clustered index on the table. Creating the clustered index, rearrange the records in an order, and then place the pages contiguously on disk.
  • Reducing Fragmentation in an Index: There are three choices for reducing fragmentation, and we can choose one according to the percentage of fragmentation:
    • If avg_fragmentation_in_percent > 5% and < 30%, then use ALTER INDEX REORGANIZE: This statement is replacement for DBCC INDEXDEFRAG to reorder the leaf level pages of the index in a logical order. As this is an online operation, the index is available while the statement is running.
    • If avg_fragmentation_in_percent > 30%, then use ALTER INDEX REBUILD: This is replacement for DBCC DBREINDEX to rebuild the index online or offline. In such case, we can also use the drop and re-create index method.
    • (Update: Please note this option is strongly NOT recommended)Drop and re-create the clustered index: Re-creating a clustered index redistributes the data and results in full data pages. The level of fullness can be configured by using the FILLFACTOR option in CREATE INDEX.

SQLAGENT CAN'T START

Error message in the event log:
 
SQLServerAgent could not be started (reason: SQLServerAgent must be able to connect to SQLServer as SysAdmin, but '(Unknown)' is not a member of the SysAdmin role).


Resolution:

Agent XPs wasn't enabled...

So need to enabled it using this script:


sp_configure 'show advanced options', 1;
GO
 RECONFIGURE;
GO
sp_configure 'Agent XPs', 1;
GO
 RECONFIGURE
GO

Sunday, 19 February 2012

Database Mail in SQL Server 2005

The SQL Mail problems, that we faced in SQL Server 7.0 and 2000, are no more. SQL Server 2005 supports and uses SMTP email now and there is no longer a need to MAPI client to send email. In SQL Server 2005, the mail feature is called Database Mail. In this article, I am going to demonstrate step-by-step, with illustrations, how to configure Database Mail and send email from SQL Server.
Database Mail has four components.
1.     Configuration Component
Configuration component has two sub components. One is the Database Mail account, which contains information such as the SMTP server login, Email account, Login and password for SMTP mail.
The Second sub component is Database Mail Profile. Mail profile can be Public, meaning members of DatabaseMailUserRole in MSDB database can send email. For private profile, a set of users should be defined.
2.     Messaging Component
Messaging component is basically all of the objects related to sending email stored in the MSDB database.
3.     Database Mail Executable
Database Mail uses the DatabaseMail90.exe executable to send email.
4.     Logging and Auditing component
Database Mail stores the log information on MSDB database and it can be queried using sysmail_event_log.
Step 1
Before setting up the Database Mail profile and accounts, we have to enable the Database Mail feature on the server. This can be done in two ways. The first method is to use Transact SQL to enable Database Mail. The second method is to use a GUI.
In the SQL Server Management Studio, execute the following statement.
use master
go
sp_configure 'show advanced options',1
go
reconfigure with override
go
sp_configure 'Database Mail XPs',1
--go
--sp_configure 'SQL Mail XPs',0
go
reconfigure 
go
Alternatively, you could use the SQL Server Surface area configuration. Refer Fig 1.0.

Fig 1.0
Step 2
The Configuration Component Database account can be enabled by using the sysmail_add_account procedure. In this article, we are going create the account, "MyMailAccount," using mail.optonline.net as the mail server and
makclaire@optimumonline.net as the e-mail account.
Please execute the statement below.
EXECUTE msdb.dbo.sysmail_add_account_sp
    @account_name = 'MyMailAccount',
    @description = 'Mail account for Database Mail',
    @email_address = 'makclaire@optonline.net',
    @display_name = 'MyAccount',
 @username='makclaire@optonline.net',
 @password='abc123',
    @mailserver_name = 'mail.optonline.net'
Step 3
The second sub component of the configuration requires us to create a Mail profile.
In this article, we are going to create "MyMailProfile" using the sysmail_add_profile procedure to create a Database Mail profile.
Please execute the statement below.
EXECUTE msdb.dbo.sysmail_add_profile_sp
       @profile_name = 'MyMailProfile',
       @description = 'Profile used for database mail'
Step 4
Now execute the sysmail_add_profileaccount procedure, to add the Database Mail account we created in step 2, to the Database Mail profile you created in step 3.
Please execute the statement below.
EXECUTE msdb.dbo.sysmail_add_profileaccount_sp
    @profile_name = 'MyMailProfile',
    @account_name = 'MyMailAccount',
    @sequence_number = 1
Step 5
Use the sysmail_add_principalprofile procedure to grant the Database Mail profile access to the msdb public database role and to make the profile the default Database Mail profile.
Please execute the statement below.
EXECUTE msdb.dbo.sysmail_add_principalprofile_sp
    @profile_name = 'MyMailProfile',
    @principal_name = 'public',
    @is_default = 1 ;
Step 6
Now let us send a test email from SQL Server.
Please execute the statement below.
declare @body1 varchar(100)
set @body1 = 'Server :'+@@servername+ ' My First Database Email '
EXEC msdb.dbo.sp_send_dbmail @recipients='mak_999@yahoo.com',
    @subject = 'My Mail Test',
    @body = @body1,
    @body_format = 'HTML' ;
You will get the message shown in Fig 1.1.

Fig 1.1
Moreover, in a few moments you will receive the email message shown in Fig 1.2.

Fig 1.2
You may get the error message below, if you haven't run the SQL statements from step 1.
Msg 15281, Level 16, State 1, Procedure sp_send_dbmail, Line 0
SQL Server blocked access to procedure 'dbo.sp_send_dbmail' of
component 'Database Mail XPs' because this component is turned off as part of
the security configuration for this server. A system administrator can enable
the use of 'Database Mail XPs' by using sp_configure. For more information
about enabling 'Database Mail XPs', see "Surface Area Configuration"
in SQL Server Books Online. 
You may see this in the database mail log if port 25 is blocked. Refer Fig 1.3.

Fig 1.3
Please make sure port 25 is not blocked by a firewall or anti virus software etc. Refer Fig 1.4.

Fig 1.4
Step 7
You can check the configuration of the Database Mail profile and account using SQL Server Management Studio by right clicking Database Mail [Refer Fig 1.5] and clicking the Configuration. [Refer Fig 1.6]

Fig 1.5

Fig 1.6
Step 8
The log related to Database Mail can be viewed by executing the statement below. Refer Fig 1.7.
SELECT * FROM msdb.dbo.sysmail_event_log

Fig 1.7

Conclusion

This article has demonstrated step-by-step instructions, with illustrations, how to configure Database Mail and send email from SQL Server.